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ABSTRACT 

This report describes the process undertaken at 
Midland Technical College (MTC) for developing and implementing an 
institutional effectiveness assessment project using student outcomes 
measures as an integral part of the assessment methodology. First, 
the report presents the background to the project, reviewing MTC's 
involvement as a model demonstration site in Project Cooperation, a 
joint assessment training effort of three educational organizations. 
Then, the report describes the process undertaken to develop a series 
of research topics that would generate data to assess the 
institution's effectiveness, including college wide discussions and 
the formation of an institutional effectiveness unit at the college. 
This section describes the four phase testing strategy using American 
College Testing instruments to assess value added cognitive growth 
among students. The next section discusses factors critical to the 
success of the project as identified by the project planning team, 
including communicating project goals and expected results to target 
constituencies and relating the assessment project to the college's 
mission. Elements described as critical to successful project 
implementation include involvement of faculty and staff in all phases 
of the project plan, and the motivation and understanding of 
students. (GFW) 
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MIDLANDS TECHNICAL COLLEGE 



ASSESSMENT OF STUDENT LEARNING AND DEVELOPMENT 
FROM ENTRY THROUGH EXIT 
AND TRANSFER TO FOUR YEAR COLLEGES 



There is a need nationwide for more accurate assessment information on the effective- 
ness of community colleges in meeting student and community needs. Valid assess- 
ment procedures and methodologies designed specifically for use by community col- 
leges to assess their effectiveness must be developed if community colleges are to be 
assessed accurately. 

Midlands Technical College has made a major commitment to institutional effective- 
ness collegewide. Even before South Carolina adopted the "Cutting Edge* 1 legis- 
lation requiring colleges to develop local effectiveness measures, President James L. 
Hudgins and college personnel were initiating a local planning and evaluation program 
designed to enhance student success and college development Mandates from the 
Southern Association of Colleges and Schools and the South Carolina Commission on 
Higher Education, along With recommendations from the AACJC Futures Commis- 
sion on "Building Communities", led to the adoption of a multidimensional Institu- 
tional Effectiveness Program, with student outcomes assessment as an integral part of 
the overall assessment methodology. 

In 1989-90, Midlands Technical College was designated as the lead institution for 
institutional effectiveness for South Carolina's two-year technical/community col- 
leges. In June 1990, the college received a third year of funding from theCommission 
on Higher Education to develop a model academic program assessment system and to 
provide leadership for faculty development, evaluation of developmental education 
programs, end dissemination of exemplary methodologies, products and practices in 
instruction. The college also received a grant from AACJC and the Kellogg 
Foundation to serve as a Beacon College for the development and replication of 
assessment methodologies and the development of institutional effectiveness pro- 
grams throughout South Carolina technical/community colleges. 
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Goals and Qfafeflfaag 



In addressing the question "How effective is our institution?'*, six broad performance 
characteristics that are of paramount importance for the ultimate success of the college 
have been identified: 

Accessible, Comprehensive Programs of High Quality 
Student Satisfaction and Retention 
Post-Education Satisfaction and Success 
Economic Development and Community Involvement 
Sound, Effective Resource Management 
Dynamic Organizational Involvement and Development 

These performance characteristics are called Critical Success Factors. As defined in 
the North Carolina Community College System Effectiveness Model, Critical Success 
Factors are "the key things that must go rightfor the system to flourish and achieve its 
goals". In order to monitor and measure college performance relative to each critical 
success factor, indicators of effectiveness and a set of measurable criteria have been 
developed for each factor. 

The indicators of effectiveness shown on page 4 provide a structure and mechanism 
for organizing the review and evaluation of Midlands Technical College's effective- 
ness in key areas. In 1990, internal assessment activities will include, but not be 
limited to: 

(1) continued refinement of the measurement criteria used to 
determine effectiveness for the College's six critical success 
factors; 

(2) continued work on the development of general education core 
competencies; 

(3) continued implementation of academic program reviews and 
the instructional effectiveness program; 

(4) full implementation of a retention and attrition study; 

(5) surveys of alumni, employers, and continuing students; 



(6) implementation of a developmental education longitudinal 
study; 

(7) ongoing work on a transfer study; 

(8) administration and analysis of an exit test (CAAP); 

(9) development of an instrument and process for assessment 
of student development; and 

( 10) assessment of library services use. 



OPERATIONALIZING INSTITUTIONAL EFFECTIVENESS 

In 1989, in order to provide an administrative structure for the institutional effective- 
ness program, the President formed an institutional effectiveness unit, named an 
Associate Vice President for Institution Effectiveness, and created an internal institu- 
tional effectiveness committee. However, the collection of information that will 
provide the data needed to determine the college's performance against a given 
standard is not assigned to a single person or unit Assessment studies and data 
collection has been decentralized and delegated to the division or department with the 
best access to the information or with the knowledge, staff, or resources to conduct the 
gathering activity. As the diagram on page five illustrates, all divisions of the college 
are involved in coordination and tracking of assessment activities to avoid duplication 
of efforts. 

The Role of Protect flnnpprflHnn 

In Spring of 1989 college personnel considered participation in Project Cooperatfon, 
a joint project of the National Council of Instructional Administrators (NCIA), the 
National Council on Student Development (NCSD) and American College Testing 
(ACT) to develop model student outcomes assessment methodologies for two-year 
colleges. After considerable investigation the decision was made to enter the project 
Project Cooperation has provided an opportunity to initiate student outcomes assess- 
ment in the general education core and to examine cognitive and affective factors that 
impact student success. Related activities in some way support most of the college' s 
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six critical success factors. The following statement was adopted as the goal of the 
student assessment program at Midlands Technical College: 



The purpose of the Midlands Technnical College student 
assessment program is to assess student learning and de- 
velopment from entry through exit and the post Midlands 
Technical College experience in order to yield information 
that will be used to enhance student learning and develop- 
ment and the overall effectiveness of the institution. 

Following extensive discussions with various college constituencies, especially aca- 
demic departments, a series of research topics, all of which support the college* s 
Critical Success Factors, were identified for Project Cooperation: 

(1) ASSET-CAAP/CAAP-CAAP Relationships that Assess Value- 
added Cognitive Growth 

(2) Assessment of Developmental Studies Program Effectiveness 

(3) Transfer Student Outcomes 

(4) Cohort Variables that Contribute to the Success of Minority and 
At-Risk Students 

(5) Appropriateness of ASSET and CAAP for Assessing Student 
Learning and Development 

Proposed outcomes of the study at Midlands Technical College include: 

( 1) establishment of a process for entry-exit testing; 

(2) assessment of the effectiveness of the Developmental Studies 
Program on students* future academic success and satisfaction; 
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(3) determination of key factors that influence the academic success 
and satisfaction of Midlands Technical College students who trans- 
fer to four-year colleges; 

(4) identification of cohort variables that contribute to the success of 
minority and at-risk students for use in improvements of programs 
and services; and 

(5) development of an improved student database that facilitates the 
educational planning process. 



CRITICAL FACTORS FOR PROJECT SUCCESS 

In order to successfully plan and initiate Project Cooperation, a project planning team 
was established. This planning team identified key challenges and critical factors for 
project success: 

(1) communicating project goals and expected educational results to 
target constituencies; 

(2) relating the assessment project to the college's mission statement 
and institutional effectiveness program; 

(3) obtaining institutional commitment; 

(4) identifying and obtaining required resources; 

(5) designing a research and evaluation plan; 

(6) training faculty and staff who would be involved; 

(7) monitoring the plan and findings, with revisions as needed; 

(8) evaluating and communicating findings; and 

(9) using results to improve student learning and development and to 
enhance institutional effectiveness. 

ERIC 7 4 n 
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lb guide the successful implementation of project criteria in support of institutional 
effectiveness, a project guide was prepared, outlining tasks, completion dates, 
person(s) responsible and associated costs for each project success factor. 

ASSESSMENT PLAN/METHODOLOGIES 

In designing the assessment plan for Project Cooperation, the characteristics and goals 
of Midlands Technical College students were considered and included in the design. 
Decisions were made to: 

(1) integrate entry assessment on CAAP as c >sely as possible into 
the instructional program. 

(2) assess aJL students in entry-level math and English classes on 
CAAP regardless of previous course work or completion of 
ASSET. Data on all entry-level students would then be available 
while allowing identification of first-time freshmen who had also 
taken ASSET in the research analysis. 

(3) conduct post-CAAP assessment upon completion of 60 quarter 
hours of credit and the general education core, since transfer stu- 
dents comprise twenty-five percent (25%) of the college popula- 
tion and few transfer students remain at Midlands Technical Col- 
lege through graduation. 

The Project Cooperation data collection plan is outlined below: 

Pre-Enrollment - Educational Planning Form 

- ASSET (Reading, Writing Skills, Writing Sample, 
Numerical Skills and Elementary Algebra) 

First Year; - CAAP Entry Testing (2 weeks into first term) 

- Student Opinion Inventory/N on-Returning Student 
Survey (after end of 3rd term) 



Second Yean - CAAP Post Testing (quarterly, beginning Fall 1990) 

- Student Opinion Inventory (with CAAP) 

- Non^Returning Student Survey (after two terms of 
non-attendance) 

- Service Utilization Data 

After Goal Completion: - Transfer Survey; Alumni Survey 

- Employment Data 

- Transfer Student Data (GPA, Performance 
in Target Courses at the four-year institution) 

- Midlands Technical College Student Tran- 
script Data (GPA, Target Course Performance, 
Retention Data) 

Data elements to be examined include: 

ASSET Educational Planning Information 

ASSET Scores (Reading/Math/Writing) 

CAAP Entry Scores (Reading/Math/Writing/Critical Thinking) 

CAAP Exit Scores (Reading/Math/Writing/Critical Thinking) 

Student Opinion Survey Data, with Local Items (midpoint and exit) 

Withdrawing/Non-Returning Student Survey Data, with Local Items 

(two terms after non-returning) 

GPA 

Grades in target courses 
Successful Goal Completion 

Developmental Studies Data (grades in target courses after 
Developmental Studies/Without Developmental Studies) 
Transfer Student Data (Grades in Target Courses/GPA) 

Relational, predictive and comparative models will be required to complete the 
analysis on the proposed research questions and achieve the desired project outcomes. 
Specific statistical applications are yet to be determined. 
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ASSESSMENT PLAN IMPLEMENTATION 



Hie aforementioned project planning team coordinated assessment implementation 
activities. This team included the Associate Vice President for Institutional Effective- 
ness, the Dean for Instructional Services, the Director of Student Assessment, and the 
Director of Research and Analysis, with the Dean for Student Entry and Enrollment 
Services serving as chairman. Joining in partnership with the major divisions of the 
college, the planning team maintained close communication and involvement with the 
college administration, Faculty Council, Str dent Assessment Committee, and individ- 
ual academic units. Most critical to the success of planning and implementation was 
the support of top administration and the involvement of faculty and students. 

Project consideration consisted of Executive Staff review, meetings with ACT repre- 
sentatives, and an examination of the relationship of the project to state requirements, 
the college's institutional effectiveness plan and college goals. The project was then 
referred for detailed analysis and an examination of its impact on the college. 

Project analysis involved extensive meetings with college personnel, a discussion 
session between ACT representatives and key faculty/staff, and an examination of the 
project's impact on resources (monetary, staffing, computing requirements and 
research). This infonnation enabled the planning team to develop a detailed project 
plan that outlined what, who, when, and the cost involved. Participation of the 
planning team at a NCSD/NCIA/ACT Summer Leadership Conference provided team 
members with the opportunity to share ideas with other colleges throughout the nation 
that were considering participation in Project Cooperation. 

Project approval involved a series of approval stages in order to assure support from 
college constituencies. Endorsements were obtained from the Student Assessment 
Committee, the Educational Affairs Council, the Student Development Management 
Council, The Faculty Council and the Executive Staff. 

Implementing Entry T*vd Assessment 

Once the decision was made to begin outcomes assessment and the research topics 
identified, the next major issue was designing and implementing strategies for entry 
level assessment Because the relationship between ASSET at entry and CAAP at exit 
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was under investigation, it was necessary to assess students on both the ASSET and 
CAAPupon entry and the CAAP again upon exit Adding to the complexity of entry- 
level assessment was the fact that Midlands Technical College was not using ASSET 
for entry assessment at the time the decision was made to participate in Project 
Cooperation. 

Challenges to entry-level assessment included: 

(1) communicating changes in entry assessment to college con- 
stituencies (stadents/faculty/staff); 

(2) defining the test population (all entering students/associate 
degree students only); 

(3) deciding to test students pre-enrollment/post-enrollment or 
a combination; 

(4) conducting in-class or out-of- class testing; 

(5) using voluntary or mandatory student participation; 

(6) determining the amount of control required for testing; 

(7) deciding who would administer the tests and conduct the 
training; 

(8) identifying how results would be disseminated (by whom 
and to whom); 

(9) determining the techniques and designs that would be estab- 
lished for mid-point and exit testing; and 

(10) deciding how results wDuld be used. 

A series of entry-level assessment approaches were considered and evaluated based 
on factors such as length of student testing time, cost, instructional time loss, impact 
on sample validity and whether or not a transition from the present entry-level 
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assessment instrument would be required. Hie options considered and associated 
problems are indicated in the table below: 



ENTRY-LEVEL ASSESSMENT APPROACHES 



Approach 


Problem 


• Current Assessment + ASSET + CAAP 


O Length 
OCost 


0 Current Assessment Pre-Enrollment; 
ASSET ft CAAP Post-Enrollment 


O Length 
OCost 

O Instructional Time Loss 


• ASSET-CAAP Pre-Enrollment 


O Length 
OCost 

O Entry Test Transition 


• ASSET Pre-Enrollment; CAAP 
Post-Enrollment (out-of-class) 


O Entry Test Transition 
O Sample Validity 


• ASSET Pre-Enrollment; CAAP 
Post-Enrollment On-class) 


O Entry Test Transition 
O Instructional Time Loss 



Because the college's Student Assessment Committee had already recommended a 
change in the entry-level assessment instrument, with ASSET as a probable replace- 
ment, the decision was made to initiate this change in July 1989, with cut-off scores 
on ASSET identified from score conversion and validation studies. Hie entry-level 
assessment model implemented at Midlands Technical College is illustrated in the 
chart below: 

MIDLANDS TECHNICAL COLLEGE 
ENTRY ASSESSMENT IMPLEMENTATION 

• ASSET Administered as Placement Test 
Prior to Enrollment 

• CAAP Administered In Entry Level Math 
and English classes 

* Math (Reading & Math) 

* English (Writing & Critical Thinking) 

erJc n 

15 



This design had the advantages of broad-based entry level assessment, a high level of 
partnerships, good student participation and the integration of assessment into the 
instructional program, all of which outweighed the disadvantages of limited instruc- 
tional time loss, administrative time loss, limited control of testing conditions and cost 
for less than complete data sets. 

Overall, this design was found to be quite effective, resulting in CAAP testing of over 
2,000 students and ASSET testing of more than 1,300 students, for a broad-based 
entry-level assessment population. The larger number of students involved in entry- 
level CAAP assessment was desired since the primary interest of the college is CAAP 
to CAAP assessment to evaluate value-added cognitive growth. A project group of 
576 took ASSET, at least two parts of the CAAP, and enrolled fall term 1989. 



FACULTY/STAFF INVOLVEMENT 

Hie single most critical ingredient for the success of Project Cooperation at Midlands 
Technical College has been the involvement of faculty and staff in all phases of the 
project plan. The planning team itself is comprised of representatives from several 
major college divisions, with these same representatives serving on the Institutional 
Effectiveness Committee, so as to yield continuity between institutional assessment 
needs and project research. Involvement activities have included the following: 

(1) College administration participated in the initial planning and 
updates. Divisional and departmental meetings with college 
faculty were held at the outset of the project to determine pri- 
orities for research needs and to allow faculty to assume "own- 
ership" of the process so that outcomes assessment would be 
viewed as non-threatening. Focus was directed to informa- 
tional needs required to allow faculty to make informed deci- 
sions regarding program improvements. This same process is 
now being implemented with Student Development staff on 
issues regarding services evaluation and related effectiveness 
criteria. 
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(2) Prior to approval of the project by the college' s Executive Staff, 
endorsements were obtained from the Educational Affairs Coun- 
cil, the Student Development Management Council, the Faculty 

. Council, and the Student Assessment Committee. 

(3) Arts and Sciences faculty have been integrally involved in the 
assessment process, since Project Cooperation focuses to a 
large extent on the evaluation of the general education core cur- 
riculum. This involvement has included input on research topics, 
designforentry-levelCAAPtestmg.designfo 
administration of CAAP entry testing in math and English classes, 
and review of ASSET results with recommendations for adjust- 
ments in cut-off scores and entry course placement 

(4) lb assess faculty sentiment regarding changes in the entry testing 
program and CAAP pilot testing, a survey was distributed in 
January 1990 to determine faculty awareness of assessment changes 
and understanding as to why these activities were being im- 
plemented. Eighty-five percent (85%) of the faculty responding 
indicated that they were aware of the changes and understood why 
these changes were being initiated; thirteen percent (13%) were 
aware of the changes but did not fully understand them; and onl, 
two percent (2%) were unaware of the changes taking place in 
student assessment 

When faculty were asked if they perceived these changes as posi- 
tive or negative, or if they required more information and data 
before they could express an opinion, an overwhelming majority 
(65%) indicated that they felt the student assessment process was 
moving in a positive direction. Hie remainder, with the exception 
of two faculty members, felt they needed more information and a 
review of student data before making a decision. The two persons 
who felt the college was moving in a negative direction voiced 
concern that changes in placement testing and entry-exit testing 
had been implemented too rapidly. 
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One of the most significant benefits of Project Cooperation has been the development 
of partnerships among major college units in support of student outcomes assessment 
for institutional effectiveness. A close working relationship has developed between 
Educational Affairs and Student Development on student assessment issues, sup- 
ported by computer programming and operations, research and analysis, the institu- 
tional effectiveness unit, and the college marketing department 



STUDENT MOTIVATION EFFORTS 

A second critical element to successful project implementation has been the motiva- 
tion and understanding of students. During initial project planning, a number of in- 
centive options were explored butdiscounted because it was feltthatthe sample would 
be skewed if entry assessment was voluntary. Midlands Technical College adopted the 
stance that student assessment is part of the instructional program, and this has been 
the message conveyed to students. A series of steps have been taken to assure student 
motivation and involvement 

(1) During entry CAAP assessment, modules were implemented in 
entry-level math and English courses based on the relationship of 
the module to the course content Writing Skills and Critical 
niinking modules were administered in English classes, while 
Math and Reading were given in math classes. 

(2) A prepared statement on the purposes of the assessment, its uses, 
and planned feedback to students, was provided to each instructor 
administering the test to assure consistency of information tc stu- 
dents. Special care was taken to let students know that they may 
rind the test difficult because it was designed to assess what they 
should know at the end of their program of study-that they should 
do their best and not feel frustrated. 

(3) To assess studentreaction, the Directorof Student Assessmentand 
Dean for Student Entry and Enrollment Services administered sev- 
eral day and evening entry-level CAAP tests and talked with stu- 
dents in a focus-group format regarding their perceptions and con- 
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cerns. Students viewed the use of assessment for college improve- 
ment as positive and desired feedback on results so they could 
assess their own progress. 

(4) To provide continued student involvement, students were sent 
letters regarding availability of their results, with a letter of thanks 
and information on the project Students would then meet indi- 
vidually with a counselor to obtain feedback on their results. Hie 
focus of feedback was on the use of results as a baseline against 
which future progress could be assessed rather than on comparison 
with norming data. 

(5) Prior to the initial implementation of CAAP post-testing in Fall 
1990, focus groups will be held with student government leaders; 
articles will be placed in the student newsletter and student news- 
paper; and letters will be sent to students congratulating them on 
their progress to date and informing them of their readiness to par- 
ticipate in CAAP post-assessment The letter will include a pre- 
paid post card to use in scheduling a test appointment along with 
a schedule of test dates. Follow-up will be provided to non- 
responders. Because CAAP post-testing will be conducted out of 
class, applicable academic-related incentive options are being ex- 
plored. 

(6) The literature provided to students on testing has been revised to 
include the purpose of student assessment and role the student 
plays in this process. Following is a summary of the statement now 
included in the student assessment brochure: 

It is the mission of Midlands Technical College to 
enable students to achieve their potential. For this 
reason, student assessment is part of the college's 
educational program. 
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From the time you apply to the college until the time 
you leave, you will participate in a series of tests and 
surveys designed to: 

(1) assess your background and academic skills 
for accurate advisement and course placement 
at entry; 

(2) obtain information on your satisfaction with 
college programs and services; and 

(3) measure gains you have made academically 
and personally while a student at the college. 

These tests and surveys will be used to help you 
achieve your individual goals and to improve college 
programs and services for all students. 

We encourage you to become a partner in the assess- 
ment and learning process. Your earnest and sincere 
participation on tests, learning tasks, exit exams and 
surveys will provide the college with accurate infor- 
mation that will be used in planning effective pro- 
grams and services. It will also help us to help you 
reach your goals. 



OTHER FACTORS OF IMPORTANCE 

Hie implementation of Project Cooperation entry-phase assessment at Midlands 
Technical College has been quite successful, particularly considering the short time 
frame between the decision to participate (June 1989) and project implementation 
(July 1990), combined with the fact that this implementation included a complete tran- 
sition in the college's entry assessment and placement program. Factors which 
contributed heavily to project success at the initial stage included: 



(1) a team approach among major divisions of the college, resulting 
in on-going student assessment partnerships; 

(2) heavy involvement of college faculty in exploration of research 
options, test administration, and project design; 

(3) consensus support of the college administration; 

(4) prior groundwork o i college mission goals and effectiveness in- 
dicators through the college's Institutional Effectiveness Program; 

(5) previous exploration of alternative placement testing options, with 
ASSET identified as one of the prime options; 

(6) integration of student assessment into the instructional program, 
with the student identified as a partner in the assessment process; 

(7) the sharing of student assessment strategies with other two-year 
colleges, especially those within Project Cooperation; and 

(8) careful project planning, with development of a project planning 
guide. 

Several concerns have become evident as the project has progressed: 

(1) Project Cooperation requires extensive communication with all 
constituencies, with considerable follow-up, organization and ad- 
ministrative detail. Care must be taken not to under -estimate cost 
in terms of faculty/staff time. 

(2) Becauseofthelargeamountofadministrativedetailanddocumen- 
tation required for entry CAAP assessment, turnaround of CAAP 
results to students and faculty did not occur as quickly as desired, 
thereby preventing in-class instructor-to-student feedback, as had 
been desired. More efficient turnaround time is required in the 
future so that this information is more meaningful to students and 
faculty. 
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(3) The appropriateness of the CAAP Math test for the total college 
population may be questionable. Initial indicators are that it may 
be most applicable to the sub-population of students planning to 
transfer to a four-year college. 



PLANNED USES OF RESULTS 

Results of student outcomes assessment are not yet available; however, the use of 
projectresearch findings will support previously identified effectiveness indicators for 
the college's critical success factors. Externally, results will be used to provide data 
to meet effectiveness criteria for state boards and commissions, accreditation agen- 
cies, and legislative delegations. Internally, the information will be used to enhance 
programs and student learning. The ultimate use of student assessment outcomes 
information is to improve student success and the overall effectiveness of the 
institution. 

Specific uses of assessment results include: 

* feedback to students and faculty; 

* evaluation of the general education core for program improve- 
ment; 

* prediction of success for transfer students, with identification of 
improvements needed; 

* validation of Developmental Studies Program effectiveness for 
underprepared students; 

* usefulness of ASSET and CAAP as approved instruments for 
Midlands Technical College students; 

* identification of high risk students for intervention; 

* utilization and satisfaction with programs and services linked to 
student needs; and 

* demonstration of institutional effectiveness to accreditation agen- 
cies, state boards, college commission and legislative delegations. 
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BENEFITS FROM PROJECT COOPERATION TO DATE 



Participation in Project Cooperation has provided Midlands Technical College with 
the opportunity to integrate a standardized outcomes assessment measure as one of the 
multiple measures of assessing student and program performance. Hie primary 
benefit is the av ailability of local and nationally normed research data that can provide 
information for use in the improvement of college programs and services that enhance 
student learning and development Spin-off benefits have included: 

(1) development of collegewide partnerships in student assessment, 
with inclusion of the student as a partner in the assessment and 
learning process; and 

(2) development of collaborative efforts with other two-year colleges 
nationwide on issues such as student assessment, institutional 
effectiveness, and student success. 
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